Image processing apparatus, image processing method, and storage medium

ABSTRACT

An image processing apparatus according to the present disclosure includes a data acquisition unit configured to acquire data in which three-dimensional medical image data and supplementary information are associated with each other, a cross section information acquisition unit configured to acquire the supplementary information from the data and acquire cross section information indicating a cross section based on the supplementary information, a cross-sectional image generation unit configured to generate cross-sectional image data corresponding to the cross section based on the three-dimensional medical image data and the cross section information, and a display control unit configured to cause a display unit to display an image based on the cross-sectional image data.

BACKGROUND Field of the Disclosure

The present disclosure relates to an image processing technique forthree-dimensional medical image data.

Description of the Related Art

There is a photoacoustic capturing apparatus that is one example of amodality apparatus that generates three-dimensional medical image data(volume data). The photoacoustic capturing apparatus can acquireinformation regarding a characteristic value of a subject as thethree-dimensional medical image data based on a photoacoustic wavegenerated due to irradiation of the subject with light.

Japanese Patent Application Laid-Open No. 2013-150787 discusses anacoustic image generation apparatus that generates and displays across-sectional image from volume data generated based on aphotoacoustic signal. Japanese Patent Application Laid-Open No.2013-150787 discusses the acoustic image generation apparatus that setsa two-dimensional plane defining the cross-sectional image according toan initial setting or an input from a user.

In a case where a desired cross section is different for each imagingoperation, it is difficult to generate the cross-sectional image of thedesired cross section by means of the initial setting. Further, as forthe method that sets the cross section defining the cross-sectionalimage by means of the input from the user, a complicated operation maybe required to set the desired cross section each time the imaging iscarried out.

SUMMARY

Some embodiments in the present disclosure are directed to providing animage processing apparatus capable of easily generating thecross-sectional image data for the three-dimensional medical image datathat corresponds to the desired cross section.

An image processing apparatus according to one aspect includes a dataacquisition unit configured to acquire data in which three-dimensionalmedical image data and supplementary information are associated witheach other, a cross section information acquisition unit configured toacquire the supplementary information from the data and acquire crosssection information indicating a cross section based on thesupplementary information, a cross-sectional image generation unitconfigured to generate cross-sectional image data corresponding to thecross section based on the three-dimensional medical image data and thecross section information, and a display control unit configured tocause a display unit to display an image based on the cross-sectionalimage data.

Further features of the various embodiments will become apparent fromthe following description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of apparatuses included in a system accordingto a first exemplary embodiment and data exchanged among theapparatuses.

FIG. 2 is a flowchart of an image processing method performed by thesystem according to the first exemplary embodiment.

FIG. 3 illustrates a camera image according to the first exemplaryembodiment.

FIG. 4 illustrates cross sections defining cross-sectional imagesaccording to the first exemplary embodiment.

FIG. 5 illustrates display data produced on a display apparatusaccording to the first exemplary embodiment.

FIG. 6 is a block diagram of apparatuses included in a system accordingto a second exemplary embodiment and data exchanged among theapparatuses.

FIG. 7 illustrates display data produced on a display apparatusaccording to the second exemplary embodiment.

FIG. 8 is a block diagram of apparatuses included in a system accordingto a third exemplary embodiment and data exchanged among theapparatuses.

FIG. 9 is a flowchart of an image processing method performed by thesystem according to the third exemplary embodiment.

DESCRIPTION OF THE EMBODIMENTS

In the following description, a first exemplary embodiment will bedescribed with reference to the drawings. FIG. 1 illustrates a blockdiagram of apparatuses included in a system according to the presentexemplary embodiment and data exchanged among the apparatuses.

The system according to the present exemplary embodiment includes amodality apparatus (medical image capturing apparatus) 110, a storageapparatus 120, an image processing apparatus 130, and a displayapparatus 140.

The modality apparatus 110 captures a subject, generatesthree-dimensional medical image data (hereinafter also referred to asvolume data) 1010 of the subject, and outputs the data 1010 to thestorage apparatus 120. The storage apparatus 120 stores the receivedvolume data 1010 therein. Information indicating a capturing portion(capturing portion information) is associated with the volume data 1010as supplementary information.

The modality apparatus 110 is an apparatus such as a photoacousticcapturing apparatus, an ultrasonic diagnostic apparatus, a magneticresonance imaging (MRI) apparatus, and an X-ray computerized tomography(CT) apparatus. In the following description, the present exemplaryembodiment will be described referring to an example in which thephotoacoustic capturing apparatus is employed as the modality apparatus110. The photoacoustic capturing apparatus is an apparatus thatgenerates information about a characteristic value corresponding to eachof a plurality of positions in the subject with use of a receptionsignal acquired when a photoacoustic wave generated due to lightirradiation is received. In other words, the photoacoustic capturingapparatus is an apparatus that generates a three-dimensional spatialdistribution of characteristic value information derived from thephotoacoustic wave as the three-dimensional medical image data.

The three-dimensional medical image data generated by the photoacousticcapturing apparatus reflects an amount and a rate of absorption of lightenergy. Examples of the three-dimensional medical image data generatedby the photoacoustic capturing apparatus include an acoustic pressure ofthe generated acoustic wave (an initial acoustic pressure), a density ofthe absorption of the light energy, a light absorption coefficient, andinformation regarding a concentration of a substance forming a tissue.The information regarding the concentration of the substance is, forexample, an oxygen saturation in blood, a total concentration ofhemoglobin, or a concentration of oxyhemoglobin or deoxyhemoglobin.Further, the information regarding the concentration of the substancemay be a glucose concentration, a collagen concentration, a melaninconcentration, a volume fraction of fat or water, or the like.

The storage apparatus 120 may be a storage medium, such as a read onlymemory (ROM), a magnetic disk, and a flash memory. Also, the storageapparatus 120 may be a storage server via a network, such as a picturearchiving and communication system (PACS).

The image processing apparatus 130 reads out the volume data 1010 fromthe storage apparatus 120, thereby acquiring the volume data 1010. Theimage processing apparatus 130 generates cross-sectional image data1030, which is subject to be displayed as a cross sectional image, basedon the volume data 1010, and outputs the data 1030 to the displayapparatus 140. The display apparatus 140 displays a cross sectionalimage based on the received cross-sectional image data 1030. The imageprocessing apparatus 130 may function as a display control unit thatcauses the display apparatus 140 to display the cross sectional imagebased on the cross-sectional image data 1030.

At this time, the image processing apparatus 130 acquires information1020 indicating a relationship between each of the plurality ofcapturing portions and a cross section. The image processing apparatus130 generates the cross-sectional image data 1030 from the volume data1010 based on the capturing portion information associated with thevolume data 1010 and the information 1020 indicating the relationshipbetween the capturing portion and the cross section. Details of a methodfor generating the cross-sectional image data 1030 will be describedbelow. For example, the information 1020 indicating the relationshipbetween the capturing portion and the cross section is constructed as alookup table indicating the relationship between each of the pluralityof capturing portions and the cross section corresponding to eachcapturing portion. In other words, the information 1020 indicating therelationship between the capturing portion and the cross section isinformation expressing the cross section predetermined for eachcapturing portion.

Owing to this configuration, a user can confirm an image correspondingto a desired cross section without performing a complicated operationfor generating the cross-sectional image data 1030 when the imageprocessing apparatus 130 reads in the volume data 1010.

Each of the apparatuses forming the system according to the presentexemplary embodiment may be realized by hardware apparatuses differentfrom one another, or all of the apparatuses may be realized by onehardware apparatus. Functions of the system according to the presentexemplary embodiment may be realized by any hardware.

A unit performing an arithmetic function in the system according to thepresent exemplary embodiment can be realized by a processor, such as acentral processing unit (CPU) and a graphics processing (GPU), or anarithmetic circuit, such as a field programmable gate array (FPGA) chip.These units may be realized not only by a single processor or arithmeticcircuit, but also by a plurality of processors or arithmetic circuits.

A unit performing a storage function in the system according to thepresent exemplary embodiment can be realized by a non-transitory storagemedium, such as a ROM, a magnetic disk, and a flash memory. Also, theunit performing the storage function may be a volatile medium, such as arandom access memory (RAM). A storage medium storing a program is anon-transitory storage medium. The unit performing the storage may berealized not only by a single storage medium, but also by a plurality ofstorage media.

A unit performing a control function in the system according to thepresent exemplary embodiment is realized by an arithmetic unit, such asa CPU. The unit performing the control function controls an operation ofeach constituent element in the system. The unit performing the controlfunction may control each constituent element in the system in responseto an instruction signal issued according to various kinds ofoperations, such as a start of the measurement, from an input unit. Inaddition, the unit performing the control function reads out a programcode stored in the storage unit and controls operation of eachconstituent element of the modality.

The display apparatus 140 is a display, such as a liquid crystaldisplay, an organic electro luminescence (EL) display, a field emissiondisplay (FED), an eyeglass-type display, and a head-mounted display. Thedisplay apparatus 140 displays the image based on the volume data, anumerical value at a specific position, and the like. The displayapparatus 140 may display a graphical user interface (GUI) for operatingthe apparatus and the image based on the volume data.

Transmission/reception of the data may be carried out via wiredcommunication or may be carried out wirelessly.

In the following description, a flow of an image processing methodperformed by the system according to the present exemplary embodimentwill be described. FIG. 2 is a flowchart of the image processing methodperformed by the system according to the present exemplary embodiment.The photoacoustic capturing apparatus according to the present exemplaryembodiment that will be described below is mainly intended for use in adiagnosis of a vascular disease, in a diagnosis of a malignant tumor ofa human or an animal, and in follow-up monitoring after a chemicaltreatment. Therefore, the photoacoustic capturing apparatus according tothe present exemplary embodiment is assumed to handle a part of a livingbody as the subject.

(Block S401: Process for Acquiring Capturing Portion Information)

The user informs the system, as an instruction, of a type of the portionof the subject to be captured with use of the input unit. The modalityapparatus 110 as a portion information acquisition unit acquires theinformation indicating the capturing portion determined based on theinstruction from the user (the capturing portion information). The usercan instruct the system to target various capturing portions, such as abreast, an arm, a palm, a back of a hand, a thigh, a shin, and an ankle,according to a purpose of the diagnosis.

An operation console including a mouse, a keyboard, and the likeoperable by the user can be employed as the input unit. Further, thedisplay apparatus 140 may be configured using a touch panel, therebyallowing the display apparatus 140 to be used as the input unit.Further, as an input method, the input unit may be configured to allowthe user to input a text or allow the user to operate the input unit toselect and input a desired item from a list on the GUI. The input unitmay be configured to allow the user to input a position that the userwants to observe and depth information. The input unit may be providedseparately from the other apparatuses.

The information that can be included in the capturing portioninformation is not only the information specifying the capturing portionitself but also any information as long as the information enablesidentification of the capturing portion. For example, in a case wherethe capturing portion can be identified considering the purpose of thediagnosis, information indicating the purpose of the diagnosis may behandled as the information indicating the capturing portion.

(Block S402: Process for Capturing Subject by Modality Apparatus)

The modality apparatus 110 as a capturing unit captures the subjectaccording to a capturing parameter based on the capturing portioninformation acquired in block S401. Examples of the capturing parameterin the photoacoustic capturing apparatus include a parameter ofirradiation light (a light intensity, a wavelength, and the like) and aparameter of a probe (a gain and the like). Besides these parameters,the modality apparatus 110 can set any kind of capturing parameter basedon the capturing portion information.

Further, in a typical case, the subject is captured in a predeterminedcapturing posture of the subject according to the capturing portion. Thecapturing posture refers to a relative positional relationship of thesubject to the modality apparatus 110, and, for example, the posture ofthe subject relative to a reference coordinate system of the apparatusis defined as illustrated in FIG. 3. For example, the modality apparatus110 may cause the display apparatus 140 to display the capturing postureas an outline on a camera image of the subject that is captured by acamera, based on the capturing portion information. An operator may setthe subject so as to align the subject with the outline displayed on thedisplay apparatus 140.

A capturing apparatus including an image sensor, such as a complementarymetal-oxide semiconductor (CMOS) sensor and a charge coupled device(CCD) sensor, which are sensitive to a visible light region, can beemployed as the camera. The camera can capture a still image and amoving image. FIG. 3 illustrates a camera image on which a posture guideaccording to the capturing apparatus is displayed being superimposed. Inthe present exemplary embodiment, the camera image will be describedreferring to an example in which the user instructs the system to targetthe back of the hand including fingers as the capturing portion for thepurpose of a diagnosis of a disease accompanying inflammation at a jointportion of a finger. An image (not illustrated) of the subject that iscaptured by the camera is displayed on a camera image 200 in real time.Further, a capturing region 201 of the modality apparatus 110 isdisplayed on the camera image 200. The capturing region 201 is a regionindicating a range where the modality apparatus 110 can capture theimage, and the capturing portion of the subject should be set withinthis region. Further, an outline image 202, which indicates thepredetermined capturing posture, is displayed being superimposed on thecamera image 200 as the guide. The outline image 202 has a guaranteedpositional relationship to the coordinate system of the apparatus. Theoperator can set the subject so as to align the image of the subjectwith the outline image 202 while viewing the camera image 200. With theaid of this function, the subject is captured in the posturecorresponding to the capturing portion.

Besides, the capturing in the capturing posture according to thecapturing portion may be realized by any other applicable method, suchas displaying a mark indicating the posture on a unit which holds thesubject.

After the subject is set in the specified posture, the modalityapparatus 110 (the photoacoustic capturing apparatus in this embodiment)irradiates the subject with pulsed light from a light irradiation unitof the modality apparatus 110. The pulsed light is absorbed by anabsorbent in the subject, and the photoacoustic wave is generated fromthe absorbent. The generated photoacoustic wave is received by areception unit of the modality apparatus 110.

The light irradiation unit includes a lit source that generates thelight and an optical system that guides the light emitted from the lightsource to the subject. The light includes pulsed light, such as aso-called square wave and triangular wave. A pulse width of the lightgenerated by the light source may be 1 ns or wider, and the pulse widthof the light generated by the light source may be 100 ns or narrower.Further, a wavelength of the light may fall within a range fromapproximately 400 nm to 1600 mm. In a case where a blood vessel iscaptured at a high resolution, the modality apparatus 110 may employ awavelength highly absorbable in the blood vessel (400 nm or longer, and700 nm or shorter). In a case where a deep portion in the living body iscaptured, the modality apparatus 110 may employ light having a typicalwavelength which is absorbable to a small extent at a background tissueof the living body (water, fat, or the like) (700 nm or longer, and 1100nm or shorter).

A laser or a light-emitting diode can be used as the light source.Further, in a case where the subject is measured using light beamshaving a plurality of wavelengths, the light source may be a lightsource which can change the wavelength. Another configuration in thecase where the subject is irradiated with the plurality of wavelengthsis a plurality of light sources that respectively generate light beamshaving wavelengths different from one another and that may alternatelyirradiate the subject. Even when the plurality of light sources is used,they will be collectively described herein as the light source. Varioustypes of lasers, such as a solid-state laser, a gas laser, a dye laser,and a semiconductor laser, can be used as the laser. A pulse laser, suchas a neodymium-doped yttrium aluminum garnet (Nd:YAG) laser and analexandrite laser, may be used as the light source. Also, a titaniumsapphire (Ti:sa) laser using Nd:YAG laser light as excitation light oran optical parametric oscillator (OPO) laser may be used as the lightsource. Also, a flash lamp or a light-emitting diode may be used as thelight source. Further, a microwave source may be used as the lightsource.

An optical element, such as a lens, a mirror, and an optical fiber, canbe used for the optical system. For example, in the case where thebreast or the like is handled as the subject, a light emission unit ofthe optical system may be formed by a diffusing plate or the like thatdiffuses the light to irradiate the subject and expand a beam diameterof the pulsed light. Also for example, in a case where a photoacousticmicroscope is used as the modality apparatus 110, the light emissionunit of the optical system may be formed by a lens or the like toirradiate the subject and focus the beam so as to increase a resolution.The light irradiation unit may directly irradiate the subject with thelight from the light source without including the optical system.

The reception unit includes a transducer that outputs an electric signalwhen the acoustic wave is received and a support member that supportsthe transducer. Further, a transducer may also be used as a transmissionunit that transmits an acoustic wave. The transducer as the receptionunit and the transducer as the transmission unit may be a single(common) transducer or may be configured as different transducers.

A piezoelectric ceramic material (e.g., lead zirconate titanate (PZT))and a polymer piezoelectric film material (e.g., polyvinylidene fluoride(PVDF)) can be used as a member forming the transducer. Additionally, anelement other than the piezoelectric element may be used. For example,an electrostatic capacitance-type transducer (a capacitivemicro-machined ultrasonic transducer (CMUT)) or a transducer using aFabry-Perot interferometer can be used. Any transducer may be employedas long as this transducer can output the electric signal when theacoustic wave is received.

(Block S403: Process for Generating Volume Data)

The modality apparatus 110 as an image data generation unit processesthe reception signal of the photoacoustic wave received by the receptionunit to generate the volume data (the three-dimensional medical imagedata) 1010. In the present exemplary embodiment, the modality apparatus110 generates the volume data 1010 associated with the capturing portioninformation that was acquired in block S401 and outputs the volume data1010 to the storage apparatus 120. The storage apparatus 120 stores thevolume data 1010 associated with capturing portion information. The dataoutput to the storage apparatus 120 may comply with, for example, theDigital Imaging and Communications in Medicine (DICOM) standard, whichis a standard specification for a medical image and communication.Three-dimensional medical image data acquired by various kinds ofmodality apparatuses is converted into the DICOM format and is storedinto a server or a storage. After that, a desired image is generated byan image processing apparatus and displayed. The DICOM file can include,in addition to the three-dimensional medical image data, metadataregarding the three-dimensional medical image data. Thethree-dimensional medical image data is recorded in a data settingsection, and the metadata is recorded in a file header section as thesupplementary information. The DICOM file is managed and operated invarious kinds of medical image management systems.

In the present exemplary embodiment, the three-dimensional medical imagedata is recorded in the data setting section of the DICOM file, and thecapturing portion information is recorded in the file header section ofthe DICOM file as the supplementary information.

As a reconstruction algorithm for converting the reception signal datainto the volume data 1010 as the three-dimensional spatial distribution,the modality apparatus 110 can employ any applicable method, such as theback projection method in the time domain, the back projection method inthe Fourier domain, and the model-based method. (the iterativecalculation method). Examples of the back projection method in the timedomain include universal back projection (UBP), filtered back projection(FBP), and Delay-and-Sum.

(Block S404: Process for Reading Out Volume Data)

The image processing apparatus 130 as an image data acquisition unitacquires the volume data 1010 by reading out the volume data 1010 storedin the storage apparatus 120. In the case where the storage apparatus120 is the PACS, the image processing apparatus 130 may acquire thevolume data 1010 from the server via the network. The user may specifythe volume data 1010 to be acquired by the image processing apparatus130 by operating the GUI displayed on the display apparatus 140.

In the present exemplary embodiment, the described system refers to theexample in which the modality apparatus 110 captures the subject andgenerates the volume data 1010, but the present exemplary embodiment maystart from the block in which the image processing apparatus 130 readsout the volume data 1010 from the storage apparatus 120. In other words,the image processing method according to the present exemplaryembodiment does not need to include the capturing of the subject by themodality apparatus 110.

(Block S405: Process for Acquiring Cross Section Information)

The image processing apparatus 130 as a cross section informationacquisition unit acquires the information indicating the cross section(the cross section information) based on the supplementary informationof the volume data 1010. The cross section information is informationused for generating the cross-sectional image data 1030 from the volumedata 1010. The cross section information is information indicating, forexample, the position and the posture of the cross section defining thecross-sectional image displayed on the display apparatus 140. The crosssection information is information indicating a coordinate of a centralposition of the cross section, a direction of the cross section, a rangeof the cross section, a slice thickness when the cross-sectional imagedata 1030 is generated, and the like. The cross section information canbe expressed by any applicable technique, such as a coordinate of thecross section and a function expressing the cross section.

In many cases, a position of a cross section desirable for observationis predetermined according to a type of the capturing portion of thesubject. For example, in the case where the subject is captured for thepurpose of diagnosis of disease accompanying the inflammation at thejoint of the finger, it is desirable that the back side of the hand isselected as the capturing portion of the subject, and a vertical crosssection along a longitudinal direction of each finger is set as thecross section.

Therefore, the image processing apparatus 130 can determine the crosssection corresponding to the capturing portion associated with thevolume data 1010 based on the capturing portion information as thesupplementary information stored in the header section of the DICOM fileand on the information indicating the relationship between the capturingportion and the cross section. In other words, the image processingapparatus 130 can acquire the cross section information indicating thecross section corresponding to the capturing portion indicated by thecapturing portion information. The information indicating therelationship between the capturing portion and the cross section may beformed by a lookup table in which each of the plurality of capturingportions, and the position and the posture of the cross sectioncorresponding to each of the capturing portions, are associated witheach other. This lookup table may be stored in the storage apparatus120, or may be stored in a storage apparatus different from the storageapparatus 120 (for example, a storage unit of the image processingapparatus 130). The image processing apparatus 130 can determine thecross section corresponding to the capturing portion by referring tothis lookup table.

For example, if a back of a right hand including fingers is specified asthe capturing portion for the diagnosis of the disease accompanying theinflammation at the joint portion of the finger in block 5401, the imageprocessing apparatus 130 can acquire cross section information 205indicating cross sections 204 corresponding to the back of the righthand including the fingers, like an example illustrated in FIG. 4. Thecross section information 205 is parameter information required togenerate cross sections of five portions that each extend along thelongitudinal direction of the finger for each the fingers and in adirection perpendicular to the palm. The cross section information 205includes parameter information indicating coordinates (x, y) of thecentral position of the cross section, an angle θ, a length l of thecross section, and a slice thickness dt with respect to the coordinatesystem of the apparatus. The length l of the cross section and the slicethickness dt may be different values for each finger, and, in a casewhere an appropriate range and thickness are predetermined for eachfinger, an appropriate value may be prepared for each piece of thecross-sectional image data 1030. The volume data 1010 corresponds to thecoordinate system of the apparatus because the volume data is capturedin the posture depending on the capturing portion, and a coordinatesystem of the cross section information 205 corresponds to thecoordinate system of the volume data 1010.

The image processing apparatus 130 as a data generation unit maygenerate volume data 1040 associated with the cross section informationacquired in block S405. Further, the image processing apparatus 130 as astorage control unit may output the volume data 1040 to the storageapparatus 120, thereby causing the storage apparatus 120 to store thevolume data 1040. The image processing apparatus 130 may cause thestorage apparatus 120 to store the volume data 1040 with the crosssection information added, by replacing the volume data 1010 alreadystored in the storage apparatus 120 with the volume data 1040.

(Block S406: Process for Generating Cross-sectional Image Data)

The image processing apparatus 130 as a cross-sectional image generationunit determines the parameters regarding the cross section, such as theposition, the direction, the range, and the slice thickness, forgenerating the cross-sectional image data based on the cross sectioninformation acquired in block 5405. The image processing apparatus 130generates the cross-sectional image data based on volume datacorresponding to the determined parameters regarding the cross section.The image processing apparatus 130 identifies the cross section in thevolume data that satisfies the position, the direction, and the rangedefined by the cross section information. Then, the image processingapparatus 130 defines a three-dimensional space (a volume) in the crosssection in the volume data is thickened by the slice thickness. Theimage processing apparatus 130 renders volume data included in thethree-dimensional space, thereby generating the cross-sectional imagedata to be displayed on the display apparatus 140. In other words, thecross-sectional image data includes not only image data expressing aspace having a thickness as thick as one voxel but also image dataexpressing a space having a thickness defined by the cross sectioninformation.

Employable rendering methods can be any applicable method, such asmaximum intensity projection (MIP), minimum intensity projection(MinIP), average intensity projection (AIP), volume rendering, andsurface rendering.

Also, the image processing apparatus 130 may generate image data otherthan the cross-sectional image data, such as image data in Which anentire region in the volume data is set as a rendering target region,from the volume data. And the image processing apparatus 130 may renderthe volume data while excluding partial data in the volume data from therendering target.

(Block S407: Process for Generating and Displaying Display Data)

The image processing apparatus 130 as the display control unit generatesthe display data to be displayed on the display apparatus 140 based onthe cross-sectional image data generated in block S406. The imageprocessing apparatus 130 outputs the generated display data to thedisplay apparatus 140, thereby causing the display apparatus 140 todisplay the display data.

FIG. 5 illustrates one example of the display data displayed on thedisplay apparatus 140. At the time of the diagnosis of the diseaseaccompanying the inflammation at the joint portion of the finger, thediagnosis can be made efficiently because respective cross-sectionalimages 301 a to 301 e of the fingers are displayed in parallel. Further,cross section position display lines 303 a to 303 e (images indicatingaxes of the fingers) may be displayed while being superimposed on anoverall image 302 (an image of the palm) of the volume data to indicatethe position, the direction, and the range of each of thecross-sectional images 301 a to 301 e in the volume data. A visual-linedirection when the overall image 302 is generated and visual-linedirections when the cross-sectional images 301 a to 303 e are generatedmay be different from each other. For example, the overall image 302 maybe a two-dimensional MIP image along a direction perpendicular to aplane displayed as the cross-sectional images 301. Further, thevisual-line directions of a plurality of cross-sectional images 301 a to301 e may be different from one another. For example, in the case wherethe capturing portion is the fingers, a pad of a thumb tends to beoriented in a direction different from pads of the other fingers whenthe palm is placed and fixed on a flat plate. Therefore, the visual-linedirections may be set in such a manner that the visual-line directionsmatch with each other among the cross sections corresponding to an indexfinger, a middle finger, a ring finger, and a little finger, while thevisual-line direction of the cross section corresponding to the thumb isset to be a direction different from the other fingers. The volume datadoes not need to be entirely displayed, but may be partially displayedas long as the displayed image extends across a range that allows theuser to understand the positions corresponding to the cross-sectionalimages 301. Further, rendering regions corresponding to each crosssection defining the plurality of cross-sectional images 301 a to 301 emay be different from one another. For example, the image processingapparatus 130 may generate the plurality of cross-sectional images 301 ato 301 e by rendering each of the five fingers with different slicethicknesses. The above-described parameters regarding the rendering togenerate the cross-sectional images 301 a to 301 e may be included inthe cross section information. Further, information which is helpful ineasily understanding the correspondences between the cross sectionposition display lines 303 and the cross-sectional images 301((a) to (e)in the case of FIG. 5) may be displayed near the cross section positiondisplay lines 303 and the cross-sectional images 301.

That is, the image processing apparatus 130 can cause the displayapparatus 140 to display thereon the overall image 302 (a first image)acquired with the visual-line direction set to a first direction.Further, the image processing apparatus 130 can cause the displayapparatus 140 to display the overall image 302 (the first image) and theplurality of cross-sectional images 301 a to 301 e (one or more secondimages) side by side, each of which is acquired with the visual-linedirection set to a second direction different from the first direction.Further, the image processing apparatus 130 can cause the imagesindicating the plurality of cross sections (the cross section positiondisplay lines 303 a to 303 e) to be displayed being superimposed on theoverall image 302. Further, the image processing apparatus 130 may causeeach of the cross-sectional images 301 to be displayed in a smaller sizethan the overall image 302. Further, the image processing apparatus 130may be configured to change the visual-line direction of the overallimage 302 or the cross-sectional images 301. The visual-line directionmay be changed to a direction determined based on an instruction fromthe user or may be changed automatically.

These displays may be automatically presented on the display apparatus140 as initial images after the user selects the volume data and theimage processing apparatus 130 reads in the volume data. Additionally,these displays may be presented on the display apparatus 140 by theuser's performing an operation for presenting these displays with use ofthe input unit. In this manner, the cross-sectional image is displayedwithout requiring the user to perform a complicated operation withrespect to specifying of the cross section information, such asspecifying the cross section position.

This configuration allows the user to easily understand what kind ofcross section each of the displayed cross-sectional images 301 indicatesin terms of the position, the direction, and the range relative to theacquired volume data of the subject.

(Block S408: Process for Determining Whether to End Display of Image)

If information indicating an end of the image display is receivedthrough the input unit from the user (YES in block S408), the imageprocessing apparatus 130 ends the image display onto the displayapparatus 140. Besides, if the image processing apparatus 130 does notreceive the instruction to end the image display (NO in block S408), theprocessing proceeds to block S409.

(Block S409: Process for Adjusting Cross Section)

The user can adjust the cross section determined in block 5405 by theinput unit. For example, the user can change the displayed crosssections by adjusting the cross section position display lines 303 onthe GUI while confirming the cross-sectional images 301 displayed on thedisplay apparatus 140. In other words, the image processing apparatus130 can update the cross section information based on the user'sinstruction to change the cross sections that is issued when thecross-sectional images 301 are being displayed. The image processingapparatus 130 can update the cross section information, such as theposition, the direction, and the slice thickness, of the cross sectionbased on the instruction from the user.

The image processing apparatus 130 may output, to the storage apparatus120, the volume data to which the cross section information after theadjustment is added. At this time, the image processing apparatus 130may cause the storage apparatus 120 to store the volume data, to whichthe cross section information after the adjustment is added, byreplacing the volume data already stored in the storage apparatus 120with it.

The present exemplary embodiment has been described referring to theexample in which the back side of the hand including the fingers is thesubject for the purpose of the diagnosis of the disease accompanying theinflammation at the joint of the finger. However, the subject is notlimited to the back side of the hand and may be another portion of ahuman body. Further, the cross section information, such as thedirection, the position, and the slice thickness of the cross section,can be varied according to a diagnosis target. For example, when a bloodvessel around a surface of the skin of an upper arm is targeted for thediagnosis, a cross section direction may be extended in parallel withthe skin to generate the cross section information, such that a positionof the cross section may be placed along a longitudinal direction of thearm defined by the capturing posture.

In the above-described manner, according to the system relating to thepresent exemplary embodiment, the cross section information can beacquired from the supplementary information associated with the volumedata. Further, according to the system relating to the present exemplaryembodiment, by generating the cross-sectional image based on the crosssection information acquired from the supplementary information, thedesired cross-sectional image can be displayed without requiring theuser to perform the complicated operation for displaying thecross-sectional image.

A second exemplary embodiment will be described, which converts thevolume data output from the modality apparatus into the cross-sectionalimage data in real time and displays the cross-sectional image in realtime.

FIG. 6 illustrates a block diagram of apparatuses included in a systemaccording to the present exemplary embodiment and data exchanged amongthe apparatuses. Configurations or data similar to the first exemplaryembodiment will be identified by the same reference numerals, anddetailed descriptions thereof will be omitted. Further, similarprocesses to the processes illustrated in FIG. 2 will be identified bythe same reference numerals, and detailed descriptions thereof will beomitted. A system in which the modality apparatus 110, the imageprocessing apparatus 130, and the display apparatus 140 are integratedmay be provided to capture the subject and display the cross-sectionalimage in real time.

In the present exemplary embodiment, the modality apparatus 110 capturesthe subject, and outputs volume data 1050 to the image processingapparatus 130. The image processing apparatus 130 receives the volumedata 1050 output from the modality apparatus 110. Methods similar toblock S402 and block S403 can be employed to capture the subject andgenerate the volume data 1050.

The image processing apparatus 130 acquires capturing portioninformation 1060 indicating the portion of the subject that is capturedby the modality apparatus 110. In the present exemplary embodiment, evenwhen the capturing portion information is not associated with the volumedata 1050 as the supplementary information, the image processingapparatus 130 acquires the capturing portion information 1060 by adifferent method.

The image processing apparatus 130 may acquire the capturing portioninformation 1060 based on an instruction from the user. For example, asillustrated in FIG. 7, the image processing apparatus 130 may cause thedisplay apparatus 140 to display a GUI 601 on which the user can selectthe capturing portion from a plurality of capturing portions with use ofthe input unit. On the GUI 601 illustrated in FIG. 7, the user canselect the capturing portion from fingers, toes, a breast, and skin.FIG. 7 illustrates an example when the user selects the fingers as thecapturing portion.

Also, the image processing apparatus 130 may acquire the capturingportion information 1060 by analyzing the volume data 1050 andidentifying the capturing portion. For example, the image processingapparatus 130 may detect a characteristic structure contained in thevolume data 1050 and acquire the capturing portion information 1060indicating a portion corresponding to this characteristic structure.Also, the image processing apparatus 130 may carry out template matchingwith a structure included in the volume data 1050 using informationindicating a characteristic structure of each capturing portion as atemplate and acquire the capturing portion information 1060 indicating acapturing portion yielding a high matching degree. Any applicable methodmay be employed to identify the capturing portion from the volume data1050.

Additionally, the image processing apparatus 130 may read out andacquire the capturing portion information 1060 indicating the capturingportion from examination order information output from a hospitalinformation system (HIS) or a radiology information system (RIS).

Subsequently, the image processing apparatus 130 acquires the crosssection information based on the capturing portion information 1060acquired in the above-described manner. A method similar to block S405according to the first exemplary embodiment can be employed as a methodfor acquiring the cross section information from the capturing portioninformation 1060.

The image processing apparatus 130 can cause the cross-sectional imageto be displayed based on the cross section information in a mannersimilar to blocks S406 and S407 according to the first exemplaryembodiment. In the present exemplary embodiment, the cross section mayalso be adjusted in a manner similar to block S409 according to thefirst exemplary embodiment.

The image processing apparatus 130 not only outputs the cross-sectionalimage data 1030 to the display apparatus 140, but also may output thevolume data 1040 associated with the acquired cross section informationto the storage apparatus 120. In other words, the image processingapparatus 130 may cause the storage apparatus 120 to store the volumedata 1040. Further, if the cross section is adjusted in block S409, theimage processing apparatus 130 may output, to the storage apparatus 120,the volume data associated with the adjusted cross section information.At this time, the image processing apparatus 130 may cause the storageapparatus 120 to store the volume data to which the adjusted crosssection information is added by replacing the volume data already storedin the storage apparatus 120 with it.

In the above-described manner, according to the system of the presentexemplary embodiment, the cross section information can be acquired fromthe acquired capturing portion information even when the supplementaryinformation is not associated with the volume data. Further, accordingto the system of the present exemplary embodiment, by generating thecross-sectional image based on the cross section information acquiredfrom the capturing portion information, the desired cross-sectionalimage can be displayed without requiring the user to perform thecomplicated operation for displaying the cross-sectional image. Further,according to the system of the present exemplary embodiment, thecross-sectional image of the desired cross section can be displayed inreal time in conjunction with the capturing of the subject.

In the following description, a third exemplary embodiment will bedescribed with reference to the drawings. The present exemplaryembodiment is different from the first exemplary embodiment in that thecross section information is acquired using a capture image generated bya capturing apparatus different from the modality apparatus. FIG. 8illustrates a block diagram of apparatuses included in a systemaccording to the present exemplary embodiment and data exchanged amongthe apparatuses. Configurations similar to the first exemplaryembodiment will be identified by the same reference numerals, anddetailed descriptions thereof will be omitted.

The system according to the present exemplary embodiment includes themodality apparatus 110, the storage apparatus 120, the image processingapparatus 130, the display apparatus 140, and a capturing apparatus 150.The modality apparatus 110 captures the subject, generates thethree-dimensional medical image data (hereinafter also referred to asthe volume data) 1050 of the subject, and outputs the data 1050 to thestorage apparatus 120. The capturing apparatus 150 is an apparatus forgenerating capture image data 1070 of the subject. An optical cameraincluding an image sensor, such as the CMOS sensor and the CCD sensorwhich are sensitive to the visible light region, may be employed as thecapturing apparatus 150.

The capture image data 1070 is an image captured for the purpose ofconfirming the position of the subject relative to the apparatus and thestate of the subject. The capture image data 1070 may also be used as areference image when the position of the subject is aligned at the timeof the capturing by the modality apparatus 110 or may be used at thetime of radiological interpretation. Therefore, a positionalcorrespondence may be established in advance between the capture imagedata 1070 generated by the capturing apparatus 150 and the volume data1050 generated by the modality apparatus 110. The capturing apparatus150 may be mounted on the modality apparatus 110 to establish thepositional correspondence between the capture image data 1070 and thevolume data 1050.

In a case where the subject has been examined by another modality inadvance and an observation target (i.e., a portion desired to beobserved) is already known from the display of the cross section, aposition of the observation target can be marked on the surface of thesubject. For example, the observation target is a tumor in a breast in adiagnosis of a breast cancer, and is a penetrating branch under the skinin a skin flap surgery for a skin graft surgery.

By marking the position of the observation target on the surface of thesubject, the position of the cross-sectional image to be generated canbe confirmed from the capture image data 1070. Marking the position ofthe observation target may reduce an influence of the capturing on themodality apparatus 110, and the marked position can be easily detectedin analyzing the capture image data 1070. For example, in the case wherethe photoacoustic capturing apparatus is employed as the modalityapparatus 110, a marking made with red ink or the like may be used.

The image processing apparatus 130 can detect the position of the markfrom the capture image data 1070. A commonly-used image processingtechnique may be used for the detection of the position of the mark, anda feature detection technique, such as edge detection processing andpattern search processing, may be employed. Further, the imageprocessing apparatus 130 determines the cross section information frominformation indicating the detected position of the mark.

The image processing apparatus 130 associates the cross sectioninformation, as the supplementary information, with thethree-dimensional medical image data, and outputs the three-dimensionalmedical image data (volume data) with the cross section information1040. From the output cross-sectional image data 1030, thecross-sectional image of the three-dimensional medical image data at theposition of the observation target is generated and displayed on thedisplay apparatus 140 based on the cross section information recorded asthe supplementary information.

In the following description, a flow for displaying the cross-sectionalimage according to the present exemplary embodiment will be describedwith reference to FIG. 9. FIG. 9 is a flowchart illustrating a methodfor determining the cross section information and a method fordisplaying the cross-sectional image that are performed by the systemaccording to the present exemplary embodiment.

In block S601, the capturing apparatus 150 captures the subject andgenerates the capture image data 1070. In the present exemplaryembodiment, the mark indicating the position of the observation target,which has been acquired from the information of another modality, isplaced on the surface of the subject.

In block S602, the image processing apparatus 130 detects the positionof the mark placed on the surface of the subject from the capture imagedata 1070 acquired in block S601, and determines the cross sectioninformation. The cross section information for generating thecross-sectional image including the observation target is determinedfrom the detected position of the mark. In S603, the determined crosssection information is output in the volume data 1040, which associatesthe cross section information with the volume data 1050.

In the present example, the system has been described referring to theexample in which the mark indicating the position of the observationtarget is placed in advance on the surface of the subject, but the crosssection information may be determined by analyzing the capture imagedata 1070 when no mark is placed. For example, when the palm isspecified as the subject by the user, the image processing apparatus 130may detect a shape of the finger from the capture image data 1070 anddetermine the cross section information.

Further, in the present exemplary embodiment, the system has beendescribed referring to the example in which the capturing apparatus 150is the optical camera provided in the photoacoustic capturing apparatus,but the capturing apparatus 150 may be different from the photoacousticcapturing apparatus and be any suitable capturing apparatus. Forexample, the capturing apparatus 150 may be an ultrasonic apparatus thattransmits an ultrasonic wave to the subject and captures subjectinformation from the reflected ultrasonic wave.

In the above-described manner, according to the system of the presentexemplary embodiment, the cross section information can be acquired fromthe capture image data obtained by the capturing apparatus that isdifferent from the modality apparatus even when the supplementaryinformation regarding the cross section information is not associatedwith the volume data. Further, according to the system of the presentexemplary embodiment, by generating the cross-sectional image based onthe cross section information acquired from the capture image data, thedesired cross-sectional image can be displayed without requiring theuser to perform the complicated operation for displaying thecross-sectional image.

Other Embodiments

Some embodiment(s) can also be realized by a computer of a system orapparatus that reads out and executes computer-executable instructions(e.g., one or more programs) recorded on a storage medium (which mayalso be referred to more fully as a ‘non-transitory computer-readablestorage medium’) to perform the functions of one or more of theabove-described embodiment(s) and/or that includes one or more circuits(e.g., application specific integrated circuit (ASIC)) for performingthe functions of one or more of the above-described embodiment(s), andby a method performed by the computer of the system or apparatus by, forexample, reading out and executing the computer-executable instructionsfrom the storage medium to perform the functions of one or more of theabove-described embodiment(s) and/or controlling the one or morecircuits to perform the functions of one or more of the above-describedembodiment(s). The computer may comprise one or more processors (e.g.,central processing unit (CPU), micro processing unit (MPU)) and mayinclude a network of separate computers or separate processors to readout and execute the computer-executable instructions. Thecomputer-executable instructions may be provided to the computer, forexample, from a network or the storage medium. The storage medium mayinclude, for example, one or more of a hard disk, a random-access memory(RAM), a read only memory (ROM), a storage of distributed computingsystems, an optical disk (such as a compact disc (CD), digital versatiledisc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memorycard, and the like.

While the present disclosure has described exemplary embodiments, it isto be understood that the claims are not limited to the disclosedexemplary embodiments. The scope of the following claims is to beaccorded the broadest interpretation so as to encompass all suchmodifications and equivalent structures and functions.

This application claims priority to Japanese Patent Application No.2017-196054, which was filed on Oct. 6, 2017 and which is herebyincorporated by reference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: a dataacquisition unit configured to acquire data in which three-dimensionalmedical image data and supplementary information are associated witheach other; a cross section information acquisition unit configured toacquire the supplementary information from data and acquire crosssection information indicating a cross section based on thesupplementary information; a cross-sectional image generation unitconfigured to generate cross-sectional image data corresponding to thecross section based on the three-dimensional medical image data and thecross section information; and a display control unit configured tocause a display unit to display an image based on the cross-sectionalimage data.
 2. The image processing apparatus according to claim 1,wherein the supplementary information includes capturing portioninformation indicating a capturing portion, wherein the cross sectioninformation acquisition unit acquires information indicating arelationship between each of a plurality of the capturing portions andthe cross section, and acquires information indicating the cross sectioncorresponding to the capturing portion, by referring to the informationindicating the relationship based on the capturing portion information,and wherein the cross-sectional image generation unit generates thecross-sectional image data based on the information indicating the crosssection corresponding to the capturing portion and the three-dimensionalmedical image data.
 3. The image processing apparatus according to claim1, wherein the supplementary information includes the cross sectioninformation, and wherein the cross section information acquisition unitacquires the cross section information from the data.
 4. The imageprocessing apparatus according to claim 1, wherein the cross sectioninformation acquisition unit acquires the cross section informationindicating a plurality of the cross sections which are different fromone another, wherein the cross-sectional image generation unit generatesa plurality of pieces of the cross-sectional image data corresponding tothe plurality of cross sections, and wherein the display control unitcauses the display unit to display the image based on the plurality ofpieces of cross-sectional image data.
 5. The image processing apparatusaccording to claim 4, wherein the cross-sectional image generation unitgenerates the plurality of pieces of cross-sectional image data byprojecting the three-dimensional medical image data corresponding toeach of the plurality of cross sections with thicknesses different fromone another.
 6. The image processing apparatus according to claim 1,further comprising: a data generation unit configured to generate datain which the three-dimensional medical image data and the cross sectioninformation are associated with each other; and a storage control unitconfigured to cause a storage unit to record the data.
 7. The imageprocessing apparatus according to claim 6, wherein the data generationunit generates the data in a DICOM format, and wherein the datageneration unit stores the cross section information in a header sectionof a DICOM file.
 8. The image processing apparatus according to claim 1,wherein the display control unit causes the display unit to display sideby side a first image of the three-dimensional medical image data thatis acquired with a visual-line direction set to a first direction and asecond image corresponding to each of a plurality of cross sectionsdifferent from one another that is acquired with the visual-linedirection set to a second direction different from the first direction,and wherein the display control unit causes an image indicating each ofthe plurality of cross sections to be displayed with this imagesuperimposed on the first image.
 9. The image processing apparatusaccording to claim 8, wherein the second direction is a directionperpendicular to the first direction.
 10. The image processing apparatusaccording to claim 8, wherein the display control unit is configured tochange at least one of the first direction and the second direction. 11.The image processing apparatus according to claim 8, wherein the firstimage is an image of a palm, the second image is a cross-sectional imageof a finger, and the image indicating each of the plurality of crosssections is an image indicating an axis of the finger.
 12. The imageprocessing apparatus according to claim 8, wherein the three-dimensionalmedical image data is image data captured by a photoacoustic capturingapparatus.
 13. An image processing method comprising: acquiring data inwhich three-dimensional medical image data and supplementary informationare associated with each other; acquiring the supplementary informationfrom the data; acquiring cross section information indicating a crosssection based on the supplementary information; generatingcross-sectional image data corresponding to the cross section based onthe three-dimensional medical image data and the cross sectioninformation; and causing a display unit to display an image based on thecross-sectional image data.
 14. The image processing method according toclaim 13, wherein the three-dimensional medical image data is image datacaptured by a photoacoustic capturing apparatus.
 15. A non-transitorystorage medium storing a program for causing a computer to performoperations comprising: acquiring data in which three-dimensional medicalimage data and supplementary information are associated with each other;acquiring the supplementary information from the data; acquiring crosssection information indicating a cross section based on thesupplementary information; generating cross-sectional image datacorresponding to the cross section based on the three-dimensionalmedical image data and the cross section information; and causing adisplay unit to display an image based on the cross-sectional imagedata.